{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Version: 12.12.2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook showcases a **recommended workflow** to prepare the modeling parameters annotated with NeuroCurator for their integration in the **Blue Brain Knowledge Graph** backed by [Blue Brain Nexus](https://bluebrain.github.io/nexus/).\n",
    "\n",
    "The four main parts of the workflow are:\n",
    "1. Prepare schemas from Neuroshapes. Publish them in Nexus.\n",
    "2. Transform source data into JSON-LD / RDF. Register them in Nexus. Recover from errors. Profile invalid data.\n",
    "3. Create test data for schema validation in Neuroshapes.\n",
    "4. Clean, _if needed_, created data in Nexus.\n",
    "\n",
    "For the features of the underlying toolkit, please refer to the [README](https://github.com/BlueBrain/nat/blob/master/kg/README.md) in the directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "mkdir repos\n",
    "cd repos\n",
    "\n",
    "git clone -b prov_literature_annotation --depth=1 https://github.com/INCF/neuroshapes.git\n",
    "git clone -b master --depth=1 https://github.com/BlueBrain/corpus-thalamus.git\n",
    "git clone -b master --depth=1 https://github.com/BlueBrain/nat.git\n",
    "\n",
    "conda create -yn kg_env python=3.7 jupyter requests\n",
    "conda activate kg_env\n",
    "pip install --extra-index-url https://testpypi.python.org/pypi pyxus==0.5.1\n",
    "\n",
    "jupyter notebook ./nat/kg/\"Literature Annotation - Knowledge Graph.ipynb\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/fonta/repos\n"
     ]
    }
   ],
   "source": [
    "cd ../.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "with open(\"TOKEN\", \"r\") as f:\n",
    "    TOKEN = f.readline().rstrip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "SCHEMA_VERSION = \"v0.2.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEPLOYMENT = \"https://bbp-nexus.epfl.ch/staging/v0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "AGENT_IRI_BASE = f\"{DEPLOYMENT}/data/thalamusproject/contributor/agent/v0.1.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEUROSHAPES_DIR = \"./neuroshapes\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANNOTATION_DIR = \"./corpus-thalamus\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "UTILS_DIR = \"./nat/kg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "sys.path.append(UTILS_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%autoreload 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "%aimport nexus_utils\n",
    "%aimport kg_utils\n",
    "%aimport parameters\n",
    "%aimport annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nexus_utils import *\n",
    "from kg_utils import *\n",
    "from parameters import *\n",
    "from annotations import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nexus client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = init_client(TOKEN, DEPLOYMENT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Commons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "commons_pipeline = PipelineConfiguration(NEUROSHAPES_DIR, client, \"neurosciencegraph\", \"commons\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "core_pipeline = PipelineConfiguration(NEUROSHAPES_DIR, client, \"neurosciencegraph\", \"core\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Literature Annotation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "organization = \"literatureannotation\"\n",
    "organization_desc = \"Data extracted from the scientific literature.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "domain = \"modelingparameter\"\n",
    "domain_desc = \"Modeling parameters extracted from the scientific literature.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = PipelineConfiguration(NEUROSHAPES_DIR, client, organization, domain, organization_desc, domain_desc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> literatureannotation\n",
      "CPU times: user 17.3 ms, sys: 4.74 ms, total: 22.1 ms\n",
      "Wall time: 314 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_organization = pipeline.create_organization()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> modelingparameter\n",
      "CPU times: user 16.5 ms, sys: 2.47 ms, total: 19 ms\n",
      "Wall time: 78 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_domain = pipeline.create_domain()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.is_organization_created()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Organization at 0x106bf4da0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_organization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.is_domain_created()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Domain at 0x106c097f0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_domain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "contexts = [\n",
    "    (\"schema\", \"v0.2.0\"),\n",
    "    (\"data\", \"v1.0.6\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> schema/v0.2.0\n",
      "<prepared> data/v1.0.6\n"
     ]
    }
   ],
   "source": [
    "prepared_contexts = core_pipeline.prepare_contexts(contexts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> schema/v0.2.0\n",
      "<already created> data/v1.0.6\n",
      "CPU times: user 35.5 ms, sys: 4.89 ms, total: 40.3 ms\n",
      "Wall time: 188 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_contexts = core_pipeline.create_contexts(prepared_contexts, publish=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_context_iri = created_contexts[\"data/v1.0.6\"].get_self_link()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://bbp-nexus.epfl.ch/staging/v0/contexts/neurosciencegraph/core/data/v1.0.6'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_context_iri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Context at 0x106c2b080>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_contexts[\"data/v1.0.6\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "core_pipeline.is_context_created(\"data\", \"v1.0.6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"schema/v0.2.0\": true,\n",
      "  \"data/v1.0.6\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "prettify(core_pipeline.are_contexts_created(contexts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> data/v1.0.6\n"
     ]
    }
   ],
   "source": [
    "prepared_data_context = core_pipeline.prepare_context(\"data\", \"v1.0.6\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> data/v1.0.6\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Context at 0x106c2bac8>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "core_pipeline.create_context(\"data\", \"v1.0.6\", prepared_data_context, publish=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Schemas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Commons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "commons_schemas = [\n",
    "    (\"parameter\", SCHEMA_VERSION),\n",
    "    (\"variables\", SCHEMA_VERSION),\n",
    "    (\"selectors\", SCHEMA_VERSION),\n",
    "    (\"annotation\", SCHEMA_VERSION),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> parameter/v0.2.0\n",
      "<prepared> variables/v0.2.0\n",
      "<prepared> selectors/v0.2.0\n",
      "<prepared> annotation/v0.2.0\n"
     ]
    }
   ],
   "source": [
    "prepared_commons_schemas = commons_pipeline.prepare_schemas(commons_schemas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> parameter/v0.2.0\n",
      "<already created> variables/v0.2.0\n",
      "<already created> selectors/v0.2.0\n",
      "<already created> annotation/v0.2.0\n",
      "CPU times: user 79.2 ms, sys: 8.48 ms, total: 87.7 ms\n",
      "Wall time: 323 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_commons_schemas = commons_pipeline.create_schemas(prepared_commons_schemas, publish=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Schema at 0x106c6ec50>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_commons_schemas[\"parameter/v0.2.0\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "commons_pipeline.is_schema_created(\"parameter\", \"v0.2.0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"parameter/v0.2.0\": true,\n",
      "  \"variables/v0.2.0\": true,\n",
      "  \"selectors/v0.2.0\": true,\n",
      "  \"annotation/v0.2.0\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "prettify(commons_pipeline.are_schemas_created(commons_schemas))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> annotation/v0.2.0\n"
     ]
    }
   ],
   "source": [
    "prepared_parameter_schema = core_pipeline.prepare_schema(\"annotation\", \"v0.2.0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> parameter/v0.2.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Schema at 0x106c47278>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "core_pipeline.create_schema(\"parameter\", \"v0.2.0\", prepared_parameter_schema, publish=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "core_schemas = [\n",
    "    (\"annotation\", SCHEMA_VERSION),\n",
    "    (\"selectors\", SCHEMA_VERSION),\n",
    "    (\"variables\", SCHEMA_VERSION),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> annotation/v0.2.0\n",
      "<prepared> selectors/v0.2.0\n",
      "<prepared> variables/v0.2.0\n"
     ]
    }
   ],
   "source": [
    "prepared_core_schemas = core_pipeline.prepare_schemas(core_schemas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> annotation/v0.2.0\n",
      "<already created> selectors/v0.2.0\n",
      "<already created> variables/v0.2.0\n",
      "CPU times: user 54.8 ms, sys: 6.1 ms, total: 60.9 ms\n",
      "Wall time: 257 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_core_schemas = core_pipeline.create_schemas(prepared_core_schemas, publish=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Literature Annotation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "schemas = [\n",
    "    (\"parameterannotation\", SCHEMA_VERSION),\n",
    "    (\"pointvalueparameter\", SCHEMA_VERSION),\n",
    "    (\"numericaltraceparameter\", SCHEMA_VERSION),\n",
    "    (\"functionparameter\", SCHEMA_VERSION),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<prepared> parameterannotation/v0.2.0\n",
      "<prepared> pointvalueparameter/v0.2.0\n",
      "<prepared> numericaltraceparameter/v0.2.0\n",
      "<prepared> functionparameter/v0.2.0\n"
     ]
    }
   ],
   "source": [
    "prepared_schemas = pipeline.prepare_schemas(schemas, is_user_domain=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<already created> parameterannotation/v0.2.0\n",
      "<already created> pointvalueparameter/v0.2.0\n",
      "<already created> numericaltraceparameter/v0.2.0\n",
      "<already created> functionparameter/v0.2.0\n",
      "CPU times: user 78.1 ms, sys: 8.21 ms, total: 86.3 ms\n",
      "Wall time: 334 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "created_schemas = pipeline.create_schemas(prepared_schemas, publish=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Raw annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "from itertools import chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO To be done with NAT in the future.\n",
    "def raw_annotations_from_dir(repo_dir):\n",
    "    path = Path(repo_dir)\n",
    "    files = path.glob(\"*.pcr\")\n",
    "    for x in files:\n",
    "        if x.stat().st_size > 0:\n",
    "            with x.open(\"r\", encoding=\"utf-8\") as f:\n",
    "                json_obj = json.load(f)\n",
    "                for y in json_obj:\n",
    "                    yield y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO To be done with NAT in the future.\n",
    "def raw_annotations_stats(raw_annotations):\n",
    "    annotated_paper_count = len(set(x[\"pubId\"] for x in raw_annotations))\n",
    "    print(annotated_paper_count, \"papers\")  # >= 113\n",
    "\n",
    "    annotation_count = len(raw_annotations)\n",
    "    print(annotation_count, \"annotations\")  # >= 582\n",
    "\n",
    "    parameter_count = len(list(chain.from_iterable(x[\"parameters\"] for x in raw_annotations)))\n",
    "    print(parameter_count, \"parameters\")  # >= 485"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "unfiltered_raw_annotations = list(raw_annotations_from_dir(ANNOTATION_DIR))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "113 papers\n",
      "582 annotations\n",
      "485 parameters\n"
     ]
    }
   ],
   "source": [
    "raw_annotations_stats(unfiltered_raw_annotations)\n",
    "# On 24.09.18:\n",
    "# 113 papers\n",
    "# 582 annotations\n",
    "# 485 parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter raw annotations - Work In Progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excluded from the raw annotations:\n",
    "\n",
    "**Annotations of facts** (i.e. no parameters) because they are currently not modeled in the SHACL shapes in Neuroshapes.\n",
    "\n",
    "Annotations with at least a **parameter of type function** because this type of parameter refers to other parameters and handling the IDs in this case is not straightforward with Nexus v0.\n",
    "\n",
    "Annotations with a **DOI containing a `:`** because Nexus v0 is expecting an URI and this is an invalid one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_selected_raw_annotation(x):\n",
    "    return (x[\"parameters\"]\n",
    "            and \"function\" not in {p[\"description\"][\"type\"] for p in x[\"parameters\"]}\n",
    "            and \":\" not in x[\"pubId\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_annotations = [x for x in unfiltered_raw_annotations if is_selected_raw_annotation(x)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "78 papers\n",
      "315 annotations\n",
      "478 parameters\n"
     ]
    }
   ],
   "source": [
    "raw_annotations_stats(raw_annotations)\n",
    "# On 24.09.18:\n",
    "# 78 papers\n",
    "# 315 annotations\n",
    "# 478 parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variable type labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO To be done through an Ontology Service serving these terms in the future.\n",
    "def variable_type_labels():\n",
    "    response = requests.get(\n",
    "        \"https://raw.githubusercontent.com/BlueBrain/nat/master/nat/data/modelingDictionary.csv\")\n",
    "    content = response.content.decode(\"utf-8\")\n",
    "    reader = csv.reader(content.splitlines(), delimiter=\";\")\n",
    "    for x in reader:\n",
    "        if x and not x[0].startswith(\"#\"):\n",
    "            yield x[0], x[2]  # type, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "variable_labels_mapping = dict(variable_type_labels())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'interbouton_distance'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variable_labels_mapping[\"BBP-121012\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "parameters = list(transform_parameters(raw_annotations, data_context_iri, variable_labels_mapping))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "427"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pv_parameters = list(select_by_type(\"nsg:PointValueParameter\", parameters))\n",
    "len(pv_parameters)\n",
    "# On 24.09.18:\n",
    "# 427"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nt_parameters = list(select_by_type(\"nsg:NumericalTraceParameter\", parameters))\n",
    "len(nt_parameters)\n",
    "# On 22.10.18:\n",
    "# 51"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO Upcoming. See filtering section.\n",
    "# f_parameters = list(select_by_type(\"nsg:FunctionParameter\", parameters))\n",
    "# len(f_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Invalid parameters - Work In Progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def condition_1(x):\n",
    "    return \"nsg:series\" not in x[\"nsg:dependentVariable\"]\n",
    "\n",
    "def condition_2(x):\n",
    "    return (\"nsg:SimpleNumericalVariable\" in x[\"nsg:dependentVariable\"][\"@type\"]\n",
    "            and \"schema:unitCode\" not in x[\"nsg:dependentVariable\"][\"nsg:series\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> condition_1 15\n",
      "<count> condition_2 1\n"
     ]
    }
   ],
   "source": [
    "invalid_pv_parameter_idxs = profile(pv_parameters, [condition_1, condition_2], flatten=True)\n",
    "# On 24.09.18:\n",
    "# <count> condition_1 15\n",
    "# <count> condition_2 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Point value parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "411"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pv_parameters) - len(invalid_pv_parameter_idxs)\n",
    "# On 24.09.18:\n",
    "# 411"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 411\n",
      "CPU times: user 15 s, sys: 1.05 s, total: 16 s\n",
      "Wall time: 2min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pipeline.create_instances(\"pointvalueparameter\", SCHEMA_VERSION, pv_parameters,\n",
    "                        exclude_idxs=invalid_pv_parameter_idxs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numerical trace parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(nt_parameters)\n",
    "# On 22.10.18:\n",
    "# 51"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 51\n",
      "CPU times: user 1.74 s, sys: 120 ms, total: 1.86 s\n",
      "Wall time: 19.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pipeline.create_instances(\"numericaltraceparameter\", SCHEMA_VERSION, nt_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO Upcoming. See filtering section.\n",
    "# len(f_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO Upcoming. See filtering section.\n",
    "# %%time\n",
    "# pipeline.create_instances(\"functionparameter\", SCHEMA_VERSION, f_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "invalid_idx = invalid_pv_parameter_idxs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "invalid_idx\n",
    "# On 24.09.18:\n",
    "# 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<error>\n",
      "{\n",
      "  \"violations\": [\n",
      "    \"Error: Violation Error(<http://www.w3.org/ns/shacl#ShapesFailed>). Node(_:ba19c2446eab6c2e395d991a3b478824) Failed property shapes Node: _:ba19c2446eab6c2e395d991a3b478824, Constraint: _:c6219e9851a259d11e5b571b3d3e1357, path: PredicatePath(<>)\",\n",
      "    \"Error: Violation Error(<http://www.w3.org/ns/shacl#minCountError>). Node(_:ba19c2446eab6c2e395d991a3b478824) MinCount violation. Expected 2, obtained: 0 Node: _:ba19c2446eab6c2e395d991a3b478824, Constraint: _:ccd251a10818a3c994f633af5ec577a1, path: PredicatePath(<https://bbp-nexus.epfl.ch/vocabs/bbp/neurosciencegraph/core/v0.1.0/series>)\"\n",
      "  ],\n",
      "  \"code\": \"ShapeConstraintViolations\",\n",
      "  \"@context\": \"https://bbp-nexus.epfl.ch/staging/v0/contexts/nexus/core/error/v0.1.0\"\n",
      "}\n",
      "<data>\n",
      "{\n",
      "  \"@context\": \"https://bbp-nexus.epfl.ch/staging/v0/contexts/neurosciencegraph/core/data/v1.0.6\",\n",
      "  \"@type\": [\n",
      "    \"nsg:PointValueParameter\"\n",
      "  ],\n",
      "  \"schema:name\": \"p-connectivity-274c8102\",\n",
      "  \"nsg:providerId\": \"274c8102-ffde-11e5-a94f-c869cd917532\",\n",
      "  \"nsg:dependentVariable\": {\n",
      "    \"@type\": [\n",
      "      \"nsg:CompoundNumericalVariable\"\n",
      "    ],\n",
      "    \"nsg:quantityType\": \"BBP-110000\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "pipeline.create_instance(\"pointvalueparameter\", SCHEMA_VERSION, pv_parameters[invalid_idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<error>\n",
      "{\n",
      "  \"violations\": [\n",
      "    \"Error: Violation Error(<http://www.w3.org/ns/shacl#ShapesFailed>). Node(_:9a265e08fbe210860ee20ec75423a482) Failed property shapes Node: _:9a265e08fbe210860ee20ec75423a482, Constraint: _:296ddc0c293f5e5646ee61a4918e1a05, path: PredicatePath(<>)\",\n",
      "    \"Error: Violation Error(<http://www.w3.org/ns/shacl#minCountError>). Node(_:9a265e08fbe210860ee20ec75423a482) MinCount violation. Expected 2, obtained: 0 Node: _:9a265e08fbe210860ee20ec75423a482, Constraint: _:5320f2ec34f2424ccd74a28548459d57, path: PredicatePath(<https://bbp-nexus.epfl.ch/vocabs/bbp/neurosciencegraph/core/v0.1.0/series>)\"\n",
      "  ],\n",
      "  \"code\": \"ShapeConstraintViolations\",\n",
      "  \"@context\": \"https://bbp-nexus.epfl.ch/staging/v0/contexts/nexus/core/error/v0.1.0\"\n",
      "}\n",
      "<data>\n",
      "{\n",
      "  \"@context\": \"https://bbp-nexus.epfl.ch/staging/v0/contexts/neurosciencegraph/core/data/v1.0.6\",\n",
      "  \"@type\": [\n",
      "    \"nsg:PointValueParameter\"\n",
      "  ],\n",
      "  \"schema:name\": \"p-connectivity-274c8102\",\n",
      "  \"nsg:providerId\": \"274c8102-ffde-11e5-a94f-c869cd917532\",\n",
      "  \"nsg:dependentVariable\": {\n",
      "    \"@type\": [\n",
      "      \"nsg:CompoundNumericalVariable\"\n",
      "    ],\n",
      "    \"nsg:quantityType\": \"BBP-110000\"\n",
      "  }\n",
      "}\n",
      "<index> 64\n",
      "<count> 0\n"
     ]
    }
   ],
   "source": [
    "pipeline.create_instances(\"pointvalueparameter\", SCHEMA_VERSION, pv_parameters, start_idx=invalid_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Nexus v0] Parameter IDs mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "462"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(parameters) - len(invalid_pv_parameter_idxs)\n",
    "# On 24.09.18:\n",
    "# 462"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 462\n",
      "<warning> More than one result page. Iteration required.\n"
     ]
    }
   ],
   "source": [
    "# Note: Need to wait a bit so that Nexus makes the instances available for retrieve_all_results().\n",
    "created_parameters_search = pipeline.instances_of_domain(resolve=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "created_parameters = pipeline.retrieve_all_results(created_parameters_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameter_uuid_mapping = dict(uuid_iri_mapping(created_parameters))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.SearchResultList at 0x107beed30>"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_parameters_search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Instance at 0x107dc2438>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "created_parameters[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://bbp-nexus.epfl.ch/staging/v0/data/literatureannotation/modelingparameter/numericaltraceparameter/v0.2.0/f647de10-ce98-4530-99c9-47b5a299b4a1'"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameter_uuid_mapping[\"90c530b4-6e05-11e6-873d-64006a4c56ef\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 411\n",
      "<warning> More than one result page. Iteration required.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.SearchResult at 0x106ce2c88>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.instances_by_schema(\"pointvalueparameter\", SCHEMA_VERSION).results[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 411\n",
      "<warning> More than one result page. Iteration required.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pyxus.resources.entity.Instance at 0x10715f048>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.instances_by_schema(\"pointvalueparameter\", SCHEMA_VERSION, resolve=True).results[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Invalid annotations because of invalid parameters - Work In Progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "invalid_pv_parameter_uuids = {pv_parameters[i][\"nsg:providerId\"] for i in invalid_pv_parameter_idxs}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_raw_annotations = [x for x in raw_annotations\n",
    "                         if not {y[\"id\"] for y in x[\"parameters\"]} & invalid_pv_parameter_uuids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(raw_annotations) - len(valid_raw_annotations)\n",
    "# On 24.09.18:\n",
    "# 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normal flow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "annotations = list(transform_annotations(valid_raw_annotations, data_context_iri, AGENT_IRI_BASE,\n",
    "                                         parameter_uuid_mapping))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "305"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(annotations)\n",
    "# On 22.10.18:\n",
    "# 305"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<count> 305\n",
      "CPU times: user 11.8 s, sys: 870 ms, total: 12.7 s\n",
      "Wall time: 2min 16s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pipeline.create_instances(\"parameterannotation\", SCHEMA_VERSION, annotations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For demonstration:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<error> Schema does not exist!\n",
      "<index> 0\n",
      "<count> 0\n"
     ]
    }
   ],
   "source": [
    "pipeline.create_instances(\"parameterannotation\", \"v0.0.0\", annotations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neuroshapes test data generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part of the workflow is intended to be done **before pushing the schemas and the data**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaning_rules = [\n",
    "    (DEPLOYMENT, \"{{base}}\"),\n",
    "    (\"thalamusproject\", \"neurosciencegraph\"),\n",
    "    (f\"{organization}/{domain}\", f\"neurosciencegraph/{organization}\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuroshapes_helper = TestDataConfiguration(NEUROSHAPES_DIR, organization, cleaning_rules)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Valid data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ParameterAnnotation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "pa_schema_name = \"parameterannotation\"\n",
    "pa_schema_version = SCHEMA_VERSION\n",
    "pa_optional_keys = [\"nsg:keyword\", \"nsg:comment\", \"nsg:experimentalProperty\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextPositionTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-all-fields-text.json\n",
      "<written> parameterannotation/v0.2.0/auto-min-fields-text.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"e6914b74-ed2a-11e5-b291-3417ebb8f5ca\", annotations),\n",
    "    pa_schema_name, pa_schema_version,\n",
    "    pa_optional_keys,\n",
    "    suffix=\"text\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FigureTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-all-fields-figure.json\n",
      "<written> parameterannotation/v0.2.0/auto-min-fields-figure.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"1132613e-31b1-11e8-b594-64006a67e5d0\", annotations),\n",
    "    pa_schema_name, pa_schema_version,\n",
    "    pa_optional_keys,\n",
    "    suffix=\"figure\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EquationTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-all-fields-equation.json\n",
      "<written> parameterannotation/v0.2.0/auto-min-fields-equation.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"c3af1a82-c5a7-11e5-b8a7-3417ebb8f5ca\", annotations),\n",
    "    pa_schema_name, pa_schema_version,\n",
    "    pa_optional_keys,\n",
    "    suffix=\"equation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TableTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-all-fields-table.json\n",
      "<written> parameterannotation/v0.2.0/auto-min-fields-table.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"5ffe9b02-ecf3-11e5-b708-64006a4c56ef\", annotations),\n",
    "    pa_schema_name, pa_schema_version,\n",
    "    pa_optional_keys,\n",
    "    suffix=\"table\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AreaTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-all-fields-area.json\n",
      "<written> parameterannotation/v0.2.0/auto-min-fields-area.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"52afbf40-04bc-11e6-b795-64006a4c56ef\", annotations),\n",
    "    pa_schema_name, pa_schema_version,\n",
    "    pa_optional_keys,\n",
    "    suffix=\"area\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PointValueParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "pvp_schema_name = \"pointvalueparameter\"\n",
    "pvp_schema_version = SCHEMA_VERSION\n",
    "pvp_optional_keys = [\"nsg:requiredTag\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SimpleNumericalVariable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> pointvalueparameter/v0.2.0/auto-all-fields-simple.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-min-fields-simple.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"870f0288-061b-11e6-9d1c-c869cd917532\", parameters),\n",
    "    pvp_schema_name, pvp_schema_version,\n",
    "    pvp_optional_keys,\n",
    "    suffix=\"simple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CompoundNumericalVariable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> pointvalueparameter/v0.2.0/auto-all-fields-compound.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-min-fields-compound.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"7516d66e-0bb4-11e6-843d-64006a4c56ef\", parameters),\n",
    "    pvp_schema_name, pvp_schema_version,\n",
    "    pvp_optional_keys,\n",
    "    suffix=\"compound\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NumericalTraceParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "nt_schema_name = \"numericaltraceparameter\"\n",
    "nt_schema_version = SCHEMA_VERSION\n",
    "nt_optional_keys = [\"nsg:requiredTag\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SimpleNumericalVariable - SimpleNumericalVariable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> numericaltraceparameter/v0.2.0/auto-all-fields-simple-simple.json\n",
      "<written> numericaltraceparameter/v0.2.0/auto-min-fields-simple-simple.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"0ebdc338-6d38-11e6-b432-64006a4c56ef\", parameters),\n",
    "    nt_schema_name, nt_schema_version,\n",
    "    nt_optional_keys,\n",
    "    suffix=\"simple-simple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SimpleNumericalVariable - CompoundNumericalVariable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([x for x in parameters\n",
    "     if \"NumericalTraceParameter\" in x[\"@type\"]\n",
    "     and \"SimpleNumericalVariable\" in x[\"nsg:dependentVariable\"][\"@type\"]\n",
    "     and \"CompoundNumericalVariable\" in {y[\"@type\"] for y in x[\"nsg:independentVariable\"]}])\n",
    "# On 24.09.18:\n",
    "# 0, no data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CompoundNumericalVariable - SimpleNumericalVariable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> numericaltraceparameter/v0.2.0/auto-all-fields-compound-simple.json\n",
      "<written> numericaltraceparameter/v0.2.0/auto-min-fields-compound-simple.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write(\n",
    "    find(\"db0e39fe-9a9c-11e6-974a-64006a4c56ef\", parameters),\n",
    "    nt_schema_name, nt_schema_version,\n",
    "    nt_optional_keys,\n",
    "    suffix=\"compound-simple\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CompoundNumericalVariable - CompoundNumericalVariable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len([x for x in parameters\n",
    "     if \"NumericalTraceParameter\" in x[\"@type\"]\n",
    "     and \"CompoundNumericalVariable\" in x[\"nsg:dependentVariable\"][\"@type\"]\n",
    "     and \"CompoundNumericalVariable\" in {y[\"@type\"] for y in x[\"nsg:independentVariable\"]}])\n",
    "# On 24.09.18:\n",
    "# 0, no data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FunctionParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "f_schema_name = \"functionparameter\"\n",
    "f_schema_version = SCHEMA_VERSION\n",
    "f_optional_keys = [\"nsg:equationParameter\", \"nsg:requiredTag\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> functionparameter/v0.2.0/auto-all-fields.json\n",
      "<written> functionparameter/v0.2.0/auto-min-fields.json\n"
     ]
    }
   ],
   "source": [
    "# TODO Upcoming. See filtering section.\n",
    "\n",
    "# Meanwhile, ad hoc and temporary code to generate the test data for Neuroshapes.\n",
    "\n",
    "fp = find(\"71233092-eb80-11e5-a9b7-64006a4c56ef\",\n",
    "          chain.from_iterable(x[\"parameters\"] for x in unfiltered_raw_annotations),\n",
    "          \"id\")\n",
    "\n",
    "tfp = transform_parameter(fp, data_context_iri, variable_labels_mapping)\n",
    "\n",
    "for i, x in enumerate(tfp[\"nsg:equationParameter\"]):\n",
    "    puuid = x[\"@id\"]\n",
    "    pbase = \"{{base}}/data/neurosciencegraph/literatureannotation/pointvalueparameter/\" + SCHEMA_VERSION\n",
    "    tfp[\"nsg:equationParameter\"][i][\"@id\"] =  f\"{pbase}/{puuid}\"\n",
    "\n",
    "neuroshapes_helper.write(\n",
    "    tfp,  # find(\"71233092-eb80-11e5-a9b7-64006a4c56ef\", parameters),\n",
    "    f_schema_name, f_schema_version,\n",
    "    f_optional_keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invalid data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ParameterAnnotation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextPositionTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-missing-entityshape-name.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-contribution.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-contributionshape-agent.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-hastarget.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-hasbody.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-selectortargetshape-hassource.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-textpositiontargetshape-hasselector.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-textpositionselectorshape-start.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-textpositionselectorshape-end.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    pa_schema_name, pa_schema_version, \"auto-min-fields-text\",\n",
    "    [\n",
    "        (\"schema:name\", \"entityshape-name\"),\n",
    "        (\"nsg:contribution\", \"contribution\"),\n",
    "        (\"nsg:contribution.prov:agent\", \"contributionshape-agent\"),\n",
    "        (\"oa:hasTarget\", \"hastarget\"),\n",
    "        (\"oa:hasBody\", \"hasbody\"),\n",
    "        (\"oa:hasTarget.oa:hasSource\", \"selectortargetshape-hassource\"),\n",
    "        # hasTarget.hasSelector: Same for the other specialized TargetShape.\n",
    "        (\"oa:hasTarget.oa:hasSelector\", \"textpositiontargetshape-hasselector\"),\n",
    "        (\"oa:hasTarget.oa:hasSelector.oa:start\", \"textpositionselectorshape-start\"),\n",
    "        (\"oa:hasTarget.oa:hasSelector.oa:end\", \"textpositionselectorshape-end\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FigureTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-missing-indexselectorshape-index.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    pa_schema_name, pa_schema_version, \"auto-min-fields-figure\",\n",
    "    [\n",
    "        # hasTarget.hasSelector.index: Same for the other specialized IndexSelectorShape (Equation and Table).\n",
    "        (\"oa:hasTarget.oa:hasSelector.nsg:index\", \"indexselectorshape-index\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AreaTarget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> parameterannotation/v0.2.0/auto-missing-fragmentselectorshape-value.json\n",
      "<written> parameterannotation/v0.2.0/auto-missing-fragmentselectorshape-conformsto.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    pa_schema_name, pa_schema_version, \"auto-min-fields-area\",\n",
    "    [\n",
    "        (\"oa:hasTarget.oa:hasSelector.rdf:value\", \"fragmentselectorshape-value\"),\n",
    "        (\"oa:hasTarget.oa:hasSelector.dcterms:conformsTo\", \"fragmentselectorshape-conformsto\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PointValueParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> pointvalueparameter/v0.2.0/auto-missing-series-simple.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-missing-variableshape-quantitytype-simple.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-missing-valueshape-value-simple.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-missing-algebraicvalueshape-unitcode-simple.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    pvp_schema_name, pvp_schema_version, \"auto-min-fields-simple\",\n",
    "    [\n",
    "        (\"nsg:dependentVariable.nsg:series\", \"series-simple\"),\n",
    "        (\"nsg:dependentVariable.nsg:quantityType\", \"variableshape-quantitytype-simple\"),\n",
    "        (\"nsg:dependentVariable.nsg:series.schema:value\", \"valueshape-value-simple\"),\n",
    "        (\"nsg:dependentVariable.nsg:series.schema:unitCode\", \"algebraicvalueshape-unitcode-simple\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> pointvalueparameter/v0.2.0/auto-missing-series-compound.json\n",
      "<written> pointvalueparameter/v0.2.0/auto-missing-variableshape-quantitytype-compound.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    pvp_schema_name, pvp_schema_version, \"auto-min-fields-compound\",\n",
    "    [\n",
    "        (\"nsg:dependentVariable.nsg:series\", \"series-compound\"),\n",
    "        (\"nsg:dependentVariable.nsg:quantityType\", \"variableshape-quantitytype-compound\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NumericalTraceParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> numericaltraceparameter/v0.2.0/auto-missing-indpendentvariable-simple.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    nt_schema_name, nt_schema_version, \"auto-min-fields-simple-simple\",\n",
    "    [\n",
    "        (\"nsg:independentVariable\", \"indpendentvariable-simple\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### FunctionParameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<written> functionparameter/v0.2.0/auto-missing-equation.json\n"
     ]
    }
   ],
   "source": [
    "neuroshapes_helper.write_missing(\n",
    "    f_schema_name, f_schema_version, \"auto-min-fields\",\n",
    "    [\n",
    "        (\"nsg:equation\", \"equation\"),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning - KNOW WHAT YOU ARE DOING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'modelingparameter'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pipeline.domain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<deprecated> 1534 instances\n",
      "CPU times: user 1min 27s, sys: 6.24 s, total: 1min 33s\n",
      "Wall time: 6min 46s\n"
     ]
    }
   ],
   "source": [
    "# %%time\n",
    "# pipeline.clean(organization=False, domain=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
